This invention relates to a technology of controlling a stream data processing system.
In recent years, there have been growing demands for a stream data processing system for receiving a large amount of data (stream data) arriving continuously, and processing the stream data in real time. For example, in a financial application for aiding stock trading, it is one of the most important tasks to promptly respond to a stock price change. When data is processed with a conventional database management system (DBMS), it has been necessary to temporarily store received stock market data in a storage device. In future, when a larger amount of stock market data is processed, it may be difficult to respond in real time to a stock price change or the like.
Individual creation of an application for processing stream data in real time leads to such problems as an elongated development period, higher development cost, and a difficulty encountered in promptly responding to a change in a task using the application. As a result, a versatile stream data processing system is in demand.
In a stream data processing system, a query is registered first, and continuously executed upon arrival of stream data. However, the stream data arrives continuously, and hence it is impossible to start a processing after waiting the arrival of all data items. In addition, the data items that have arrived at the system need to be processed in the order of arrival without being affected by the load of data processing.
In the technology disclosed in R. Motwani, J. Widom, A. Arasu, B. Babcock, S. Babu, M. Datar, G. Manku, C. Olston, J. Rosenstein, and R. Varma, “Query Processing, Resource Management, and Approximation in a Data Stream Management System”, In Proc. of the 2003 Conf. on Innovative Data Systems Research (CIDR), January 2003, there is introduced a notion called sliding window (hereinafter referred to as “window”) to implement a real-time processing while cutting out a part of the stream data by specifying a time range such as the last 10 minutes or a number range such as the latest 1,000 data items.
As a language for describing a query for obtaining data, continuous query language (CQL) capable of specifying a window is disclosed in above-described “Query Processing, Resource Management, and Approximation in a Data Stream Management System”. The CQL is an extended version of structured query language (SQL) which is used widely in the DBMS, and is capable of specifying a window. A technology of specifically using the CQL or the like is disclosed in, e.g., JP 2006-338432 A.
However, the stream data arrives continuously, and thus, even when the processing behavior of the stream data processing system is to be changed based on the received data, it is impossible to specify data serving as a delimiter. On the other hand, a transmitter of the stream data, e.g., a stream data transmitting application recognizes data serving as a delimiter. Accordingly, as a technology for recognizing a delimiter of the stream data, a technology of determining a target of a processing (decode processing) via a packet filter based on packet identifiers added to packets forming the stream data (image according to MPEG or the like) is disclosed in JP 2003-298661 A.